Document isolated applications, branched databases, and per-PR CI previews - #715

Stacked on Deploying from CI: lead with OIDC, and correct the deploy reference where it disagrees with Harper. The base branch is #713's branch, so this diff shows only the changes added here.

⊙ Problem

Harper 5.3.0 shipped two keys for an application's root-config entry, isolated (harper#2524) and branchedDatabases (harper#2352, #2523, #2426, #2517), plus a threads.maxIsolated setting. None of them has reference docs. The multiple-applications guide says every co-located application shares the worker threads, which is no longer true for an isolated application. The 5.3 release notes have no entry for isolated application threads.

The CI guide also needs a recipe for isolated deployments and a preview for each PR. These previews deliberately share the cluster databases; database forks are a later enhancement.

Checking the docs against a local 5.3.0 turned up a bug: an application deployed by package or by_ref ignores branchedDatabases and reads and writes the base databases. It is filed as A package deploy's branchedDatabases is ignored at load, so the application runs on its base databases, and a fix is dispatched.

The per-PR recipe is included with shared databases, following the maintainer's decision. It omits branchedDatabases while harper#3071 is open and explains that preview writes, schemas, instance-wide roles, and background work affect the shared instance. Fabric host registration and private-source credential grants are explicit prerequisites.

❓ Your call: The reference documents manual steps around harper#3071: payload-deploy a new application without restarting, add branchedDatabases to its entry in harper-config.yaml on every node, restart, and check each node's fork directory. I verified this on one node. On a cluster, a node missed in step 2 writes the base, and those writes replicate, so step 2 says so in bold. Publish these steps, or say only that a package entry doesn't work yet?

❓ Your call: The brief asked the release note to cite host-manager#224. That repo is internal, so a public reader would get a 404. The release note and the reference say what Fabric's proxy does, without the link.

💡 Solution

• deploying-from-ci.mdx adds isolated deployment options for the existing release workflows and a per-PR workflow using pinned head commits, unique projects/hosts, and cleanup on closure or retargeting. It uses a default-branch-pinned OIDC policy and no PR checkout, reconciles current PR state and base, cleans each configured node independently with a presence check and worker-aware restart, verifies removal, and reports failed targets. It documents direct node URLs and their OIDC audiences, approval-gated cleanup, routing, capacity, private-source provisioning, shared data, and the later branching fix.

• A new reference section in reference/components/applications.md covers:

  • where both keys live, and what a deploy can set;
  • for isolated applications: how a request reaches one, what admission requires and refuses, and restart and drop behavior;
  • for branched databases: what the fork is, how code reaches it, what fails the load, and how a fork is removed.

  Look hardest at Reaching an isolated application. It states the routing caveat from harper#2757: a request to Harper's own ports, even one addressed to the application's host, is answered by the shared workers. Also check the known-issue warning for harper#3071.

• threads.maxIsolated in options.md: the default of 8, the 409 past the cap, and that dedicated threads are added to threads.count and count when Harper divides memory into the default heap limit (which has a 512 MB floor). Also tls.unixDomainSockets, which isolation requires and which had no entry.

• multiple-applications.mdx now says that worker threads are shared unless an application is isolated, and that an isolated application's exports don't collide with the shared workers' exports. A new subsection, A private fork of a database, sits under the namespacing section.

• 5.3.md gets an Isolated Applications entry next to Branched Databases, and a known-issue paragraph for harper#3071 and harper#3044.

• reference/components/javascript-environment.md distinguishes shared data from per-worker API objects and fork-aware imports.

• reference/components/module-loading.md qualifies shared globals and fork resolution under the default loader.

• reference/http/overview.md limits export sharing to applications loaded in the same worker.

• reference/operations-api/operations.md states restart/drop scope and cluster cleanup limits, documents the v5.3.0 restart_service.scope option and the thread-response fields used by cleanup.

• reference/components/plugin-api.md qualifies plugin startup and shutdown as occurring in workers that load the application.

• reference/components/extension-api.md qualifies resource and protocol extension callbacks for isolated applications.

⚖️ Alternatives

❓ Your call: Review depth checkpoint after 3 counted full passes: the latest full review took about 21 minutes and exposed one accepted cleanup major plus approval, fork-refresh, and GitHub event-policy gaps. At that checkpoint, the diff grew from 375 to 432 changed lines across 11 documentation files, with no runtime code changes. I chose continue for one final complete-artifact review because node-bound cleanup changes the workflow's failure behavior; splitting the recipe would leave that fix unreviewed. Manual per-PR commands remain the smaller alternative, but do not provide the requested CI lifecycle recipe.

❓ Your call: Post-review delta: 10 changed lines in 3 files, not re-reviewed — only prose clarifications requested by the final review: worker-local objects/caches, replicated policy setup, and approval requests for never-previewed closures. The workflow shell is identical to the complete artifact reviewed at ad447fdb79c6; the final full round found no blocker or major and discharged its closing-round obligation. SNI rewrite checks and retained-install source evidence below answer the remaining verification questions.

• One project and hostname per PR keeps concurrent previews separately addressable. A fixed pool of preconfigured slots would reduce routing and credential setup but requires allocation and can evict another PR on collisions. This guide uses the existing deploy/drop lifecycle, documents manual prerequisites, and provides cleanup recovery. Direct node endpoints and OIDC audiences are required for cluster cleanup; a balanced endpoint cannot establish absence on each node.
• One approval environment covers deploy and cleanup. Every same-repository closure is admitted so cleanup can recover from a missed retarget run, including PRs that never had a preview. With required reviewers, these can still ask for approval; an absent preview is skipped after approval. A separate automatic cleanup workflow/environment with a drop-only role would reduce approval noise but adds policy and lifecycle setup.
• The preview workflow runs from the trusted default branch. pull_request_target plus workflow_ref pinned to main prevents a PR from changing the privileged runner workflow. It never checks out PR code. The source preview still executes that code on Harper, so the guide limits the recipe to trusted same-repository PRs.
• The summaries leave out an edge case on purpose. The guide and the release note say a redeploy that keeps an app isolated restarts only its own thread. They do not repeat the exception: retrying an activation whose release is already live restarts every worker. The reference's restart list carries that exception, and both summaries link to it. The review raised this three times. Copying the exception into every summary is the alternative; it adds a fourth place to keep in step.
• The harper#3071 caveat is in three places (reference, guide, release note), not one warning that the others link to. A reader of the release note or the guide needs to see it without following a link. A comment on harper#3071 points whoever fixes it back to this PR.

Product and architecture tour

One preview per PR, with shared data

What happens when a PR opens, changes, or closes?

Preview lifecycle

1. Prepare access — Register the hostname and source grant; configure direct node URLs and a matching OIDC audience for each, pinned to the trusted workflow.
2. Open or update — Read the current PR head and deploy it as my-app-pr-N in a dedicated worker.
3. Browse — Route the preview hostname to its Unix socket; the preview uses the same database as the base app.
4. Close, merge, or retarget — Approve cleanup if the environment requires it. Check each node, drop only if present, restart only a live preview worker, and verify removal. Failed targets are reported; shared writes remain.

Preview worker state and exported routes are separate. Database writes, schema changes, roles, and background work still affect the shared instance. A private fork is a future extension after harper#3071 is fixed.

Separate workers, shared database

The proxy chooses an application's worker, while each application reaches the shared databases.

flowchart LR
  MainHost[Main application host] --> Proxy[Proxy]
  PreviewHost[PR preview host] --> Proxy
  Proxy --> MainWorker[Main application workers]
  Proxy --> PreviewWorker[Dedicated PR worker]
  MainWorker --> Data[(Shared databases)]
  PreviewWorker --> Data

Worker isolation and database branching are independent

Which behavior should an operator rely on today?

The reference documents worker admission, routing, restart scope, and cleanup separately from database forks. threads.maxIsolated bounds dedicated workers. On 5.3.0 and 5.3.1 package deploys can isolate workers but ignore branchedDatabases; the current manual payload workaround remains documented with its per-node caveat.

• CI preview lifecycle — The trusted workflow deploys and removes the unique PR project, checking each node, skipping absent previews and verifying component/worker removal.
• Isolation and branching reference — Defines their separate behavior and prerequisites.

Where each claim comes from (Harper v5.3.0, 726dd1e)

✅ Verification

• npm run format:check: clean. npm run build: passes.
• I ran the commands against harper@5.3.0 from npm, following AGENTS.md: a scratch HOME and ROOTPATH, threads.count: 2, threads.maxIsolated: 1, http.securePort set, and tls.unixDomainSockets: true.

Additional CI-guide validation (Harper 5.3.0):

• A scratch install ran the new preview-role setup and pinned by_ref deployments against a local git fixture. Preview deploy/drop and both cleanup reads worked with the four-operation allowlist. The final workflow_ref / pull_request_target trust policy was also accepted by Harper 5.3.0.
• A write through preview 42's socket was visible through the base application's shared HTTP endpoint. Redeploying preview 42 changed only its worker id; shared worker ids and preview 43's worker id stayed unchanged. Dropping preview 42 removed its worker and preserved those other ids.
• The documented workflow YAML parsed; 14 final extracted-shell checks covered pinned deployment, closure, retargeting, mixed node state, a worker retained after a drop, continuing after CLI failure, failure summaries, files without a worker, malformed responses, empty node targets, and invalid PR numbers.
• A cleanup retry after directory removal stopped the remaining preview worker; repeating cleanup preserved shared and sibling worker ids. An SNI TLS array with two certificate entries produced isolated sockets when the root tls_unixDomainSockets setting was enabled.
• The final workflow ran against two independent Harper Core 5.3.0 instances. Node A had no preview while node B did: cleanup removed B's preview without replacing either shared pool or A's sibling preview worker. Repeated cleanup and files-only cleanup preserved the remaining worker ids. A node-specific OIDC audience policy was accepted; the live token exchange was not exercised.
• Additional SNI persistence check on 5.3.0: the two-entry TLS array and root tls_unixDomainSockets flag survived set_configuration logging_level=warn replicated=false and a HARPER_CONFIG merge/restart; the isolated worker and socket were present afterward without the legacy environment flag.
• Retained-install cleanup source: whole-component drop calls retireComponentDirectory(...).discard() before the restart decision (operations.js:1549-1575); discard prunes dormant builds and reclaims deployment records, with failures logged (Application.ts:3871-3916). This also covers a files-only restart=false drop; journaled records may remain.
• Release attribution: fetched core tags and matched local v5.1.0^{commit} to git ls-remote origin refs/tags/v5.1.0 at 001bf7b9c55963f7dcd938087acd0047d19b8a62; the tagged source defines the socket setting. The isolated/branching v5.3.0 surface was likewise checked against the fetched release tag.
• npm run format:write, npm run format:check, and npm run build passed. The review CLI's generic recovery check reports no-restart-test because it requires a touched repository test; this docs repository has no automated suite. The scratch restart/cleanup checks above are external validation, not committed tests.

Not exercised end to end: GitHub's live OIDC exchange, live Fabric DNS/TLS routing, private-source grants on Pro/Fabric, or Pro multi-node replication. The two-instance cleanup check used independent Core instances with direct node calls, not a replicated cluster. GitHub trigger and environment behavior were checked against GitHub's event reference.

Not verified locally (source only):

• Fabric's routing to an isolated application's socket. That needs a Fabric cluster.
• A self-managed proxy in front of the socket. I tested the socket only with curl --unix-socket.
• Behavior across nodes: one fork per node, a replicated drop removing each node's fork, and peers skipping admission. Core alone has no replication.
• The LMDB, native loader, empty blobPaths, 250-character name, and damaged-fork failures.

Refs harper#642, harper#3071

Original reference work: Claude Opus 5.5. CI preview guide and consistency fixes added by Codex in this revision.

Related PRs: #713 overlaps, #671 independent, #676 independent, #710 independent, #679 independent, #683 independent, #691 independent, #697 independent, #709 independent, #712 independent, #707 independent, #714 independent

_{Dispatch: task documentation-branched-isolated-preview-deploys · queued by unknown · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=unknown; ran=none; rounds=1 @ f9ce6ad}

_{Review-Attention: deep ~25m (raised: unmeasured diff) @ f9ce6ad}